Skip to content

Conversation

@Iroy30
Copy link

@Iroy30 Iroy30 commented Jun 4, 2025

Fixes #3626

Summary/Motivation:

Add cuOpt math optimization (includes LP and MILP) solver backend to Pyomo so users can solve pyomo models with cuOpt

Changes proposed in this PR:

  • Add cuopt_direct solver plugin

Legal Acknowledgement

By contributing to this software project, I have read the contribution guide and agree to the following terms and conditions for my contribution:

  1. I agree my contributions are submitted under the BSD license.
  2. I represent I am authorized to make the contributions and grant the license. If my employer has rights to intellectual property that includes these contributions, I represent that I have received permission to make contributions and grant the required license on behalf of that employer.

@codecov
Copy link

codecov bot commented Jun 17, 2025

Codecov Report

❌ Patch coverage is 77.23577% with 56 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.01%. Comparing base (38ff2e1) to head (662d2ec).

Files with missing lines Patch % Lines
pyomo/solvers/plugins/solvers/cuopt_direct.py 77.23% 56 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3620      +/-   ##
==========================================
- Coverage   89.24%   86.01%   -3.23%     
==========================================
  Files         907      908       +1     
  Lines      104866   105112     +246     
==========================================
- Hits        93585    90412    -3173     
- Misses      11281    14700    +3419     
Flag Coverage Δ
builders 29.18% <19.10%> (-0.02%) ⬇️
default 85.84% <77.23%> (?)
expensive 35.70% <19.10%> (?)
linux ?
linux_other ?
osx ?
win ?
win_other ?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link
Contributor

@mrmundt mrmundt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are working on making cuopt available in our testing infrastructure; can you please add tests to this PR?

@mrmundt
Copy link
Contributor

mrmundt commented Jul 1, 2025

@Iroy30 - We've been able to make cuopt available on our internal testing machines. Can you please add tests to this PR?

@Iroy30 Iroy30 force-pushed the add_cuopt_direct_solver_plugin branch from 5dbf9dd to 6f64094 Compare July 2, 2025 12:33
@Iroy30
Copy link
Author

Iroy30 commented Jul 2, 2025

@mrmundt Thanks! We have added tests by enabling testing cuopt with LP and MILP capabilities in tests/solvers.py. Let us know if:

  1. There is any other testing that needs to be enabled.
  2. For testing cuOpt backend in CI, it needs a GPU, is that something pyomo team intends to add or will it be disabled in github CI?

The following is the testing output I get relevant to cuOpt

Testing scenario generation
------------------------------
LP_block, cuopt, python
   ok: 
LP_block_kernel, cuopt, python
   ok: 
LP_compiled, cuopt, python
   ok: 
LP_compiled_dense_kernel, cuopt, python
   ok: 
LP_compiled_sparse_kernel, cuopt, python
   ok: 
LP_constant_objective1, cuopt, python
   ok: 
LP_constant_objective1_kernel, cuopt, python
   ok: 
LP_constant_objective2, cuopt, python
   ok: 
LP_constant_objective2_kernel, cuopt, python
   ok: 
LP_duals_maximize, cuopt, python
   ok: 
LP_duals_maximize_kernel, cuopt, python
   ok: 
LP_duals_minimize, cuopt, python
   ok: 
LP_duals_minimize_kernel, cuopt, python
   ok: 
LP_inactive_index, cuopt, python
   ok: 
LP_inactive_index_kernel, cuopt, python
   ok: 
LP_infeasible1, cuopt, python
   ok: 
LP_infeasible1_kernel, cuopt, python
   ok: 
LP_infeasible2, cuopt, python
   ok: 
LP_infeasible2_kernel, cuopt, python
   ok: 
LP_piecewise, cuopt, python
   ok: 
LP_piecewise_nosuffixes, cuopt, python
   ok: 
LP_simple, cuopt, python
   ok: 
LP_simple_kernel, cuopt, python
   ok: 
LP_trivial_constraints, cuopt, python
   ok: 
LP_trivial_constraints_kernel, cuopt, python
   ok: 
LP_unbounded, cuopt, python
   ok: 
LP_unbounded_kernel, cuopt, python
   ok: 
LP_unused_vars, cuopt, python
   ok: 
LP_unused_vars_kernel, cuopt, python
   ok: 
MILP_discrete_var_bounds, cuopt, python
   ok: 
MILP_discrete_var_bounds_kernel, cuopt, python
   ok: 
MILP_infeasible1, cuopt, python
   ok: 
MILP_infeasible1_kernel, cuopt, python
   ok: 
MILP_simple, cuopt, python
   ok: 
MILP_simple_kernel, cuopt, python
   ok: 
MILP_unbounded, cuopt, python
   ok: 
MILP_unbounded_kernel, cuopt, python
   ok: 
MILP_unused_vars, cuopt, python
   ok: 
MILP_unused_vars_kernel, cuopt, python
   ok: 

Comment on lines 60 to 62
t0 = time.time()
self.solution = cuopt.linear_programming.solver.Solve(self._solver_model)
t1 = time.time()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is fine, but just so you're aware, we have this lovely little utility called TicTocTimer that you may want to consider using: https://pyomo.readthedocs.io/en/latest/api/pyomo.common.timing.TicTocTimer.html

@whart222
Copy link
Member

whart222 commented Jul 8, 2025

@mrmundt What do you think about including this solver interface in pyomo.contrib.solvers? Would it make sense to pull-in new solver interfaces there, since that's where the new solver API is evolving?

@mrmundt
Copy link
Contributor

mrmundt commented Jul 8, 2025

@mrmundt What do you think about including this solver interface in pyomo.contrib.solvers? Would it make sense to pull-in new solver interfaces there, since that's where the new solver API is evolving?

@whart222 - I am evenly split. Because we are still messing with what the new solver interfaces are going to actually do / how they will handle input and present output, I don't know if we want to put "new" solvers there or just "well-established" ones that we can robustly test / really know what they are supposed to do and return.

@mrmundt
Copy link
Contributor

mrmundt commented Jul 8, 2025

@Iroy30 - I forgot to post this last week, but all of the failures are of the variety:

self = <pyomo.solvers.tests.checks.test_no_solution_behavior.Test_LP_unbounded_kernel testMethod=test_cuopt_python>

    def return_test(self):
>       return failed_solve_test(self)

/tester/python/python39/pyomo/pyomo/solvers/tests/checks/test_no_solution_behavior.py:92: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/tester/python/python39/pyomo/pyomo/solvers/tests/checks/test_no_solution_behavior.py:61: in failed_solve_test
    opt, results = model_class.solve(
/tester/python/python39/pyomo/pyomo/solvers/tests/models/base.py:123: in solve
    results = opt.solve(
/tester/python/python39/pyomo/pyomo/opt/base/solvers.py:99: in solve
    self._solver_error('solve')
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <pyomo.opt.base.solvers.UnknownSolver object at 0x7f86ad9ab040>
method_name = 'solve'

        def _solver_error(self, method_name):
>           raise RuntimeError(
                """Attempting to use an unavailable solver.
    
    The SolverFactory was unable to create the solver "%s"
    and returned an UnknownSolver object.  This error is raised at the point
    where the UnknownSolver object was used as if it were valid (by calling
    method "%s").
    
    The original solver was created with the following parameters:
    \t"""
                % (self.type, method_name)
                + "\n\t".join("%s: %s" % i for i in sorted(self._kwds.items()))
                + "\n\t_args: %s" % (self._args,)
                + "\n\toptions: %s" % (self.options,)
            )
E           RuntimeError: Attempting to use an unavailable solver.
E           
E           The SolverFactory was unable to create the solver "cuopt"
E           and returned an UnknownSolver object.  This error is raised at the point
E           where the UnknownSolver object was used as if it were valid (by calling
E           method "solve").
E           
E           The original solver was created with the following parameters:
E           	executable: cuopt
E           	solver_io: python
E           	type: cuopt
E           	_args: ()
E           	options: {}

/tester/python/python39/pyomo/pyomo/opt/base/solvers.py:116: RuntimeError

@mrmundt
Copy link
Contributor

mrmundt commented Jul 8, 2025

@Iroy30 - Two more things:

  1. Please run black -S -C on your files
  2. Something in our testing environment isn't loading correctly. I'm looking into it.

@Iroy30
Copy link
Author

Iroy30 commented Oct 17, 2025

@jsiirola The segfaults you see in CI with 25.08 is concerning. Was the CI running the 25.08 release or the daily build of cuopt(this could cause the segfault). If 25.08 release was causing segfault then we might have to enforce minimum cuopt 25.10 and meanwhile I will debug 25.08 on my end because we were not seeing this previously

@jsiirola
Copy link
Member

Was the CI running the 25.08 release or the daily build of cuopt (this could cause the segfault). If 25.08 release was causing segfault then we might have to enforce minimum cuopt 25.10 and meanwhile I will debug 25.08 on my end because we were not seeing this previously

The CI is running cuopt-cu12/25.8.0 from PyPI. I have added 25.10.0 (also from PyPI), but that won't make it to the build machines for testing until later tonight (it is waiting on finalizing the Pyomo 6.9.5 release)

@jsiirola
Copy link
Member

We attempted to upgrade cuopt-cu12 to 25.10.0 (on a machine running RHEL9.6). The packages installed, but importing cuopt fails:

>>> import cuopt
.local/lib/python3.11/site-packages/libcuopt/load.py:95: RuntimeWarning: Failed to load libcuopt library: libcuopt.so. Error: .local/lib/python3.11/site-packages/libcuopt/lib64/libcuopt.so: undefined symbol: _ZSt21__glibcxx_assert_failPKciS0_S0_. Falling back to relying on system loader. cuOpt functionality may be unavailable. This might lead to a generic error such as 'libcuopt.so missing' if the library cannot be found.
  warnings.warn(
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".local/lib/python3.11/site-packages/cuopt/__init__.py", line 24, in <module>
    from cuopt import linear_programming, routing
  File ".local/lib/python3.11/site-packages/cuopt/linear_programming/__init__.py", line 16, in <module>
    from cuopt.linear_programming import internals
  File ".local/lib/python3.11/site-packages/cuopt/linear_programming/internals/__init__.py", line 30, in <module>
    from cuopt.linear_programming.internals.internals import (
ImportError: libcuopt.so: cannot open shared object file: No such file or directory

@Iroy30
Copy link
Author

Iroy30 commented Oct 22, 2025

It could be because it pulls some old artifacts and gives a blanket error. Could you try the following:

pip uninstall -y cuda-python cuda-bindings cuda-core
pip install --upgrade --extra-index-url=https://pypi.nvidia.com/ cuopt-cu12 nvidia-nvjitlink-cu12 rapids-logger==0.1.19

@bbozkaya
Copy link

@mrmundt, @jsiirola would it help communicate and resolve faster if we had a Slack channel set up for communication?
If yes, please provide an email address to which we can send a Slack join request.

@mrmundt
Copy link
Contributor

mrmundt commented Oct 23, 2025

Hi @bbozkaya - It would be best if you attended one of our weekly dev calls which happen every Tuesday at 12:30PM MT. Please email [email protected] and we can send you the connections details.

@Iroy30
Copy link
Author

Iroy30 commented Oct 28, 2025

@jsiirola Did the instructions fix the cuOpt import?

def __init__(self, **kwds):
kwds["type"] = "cuoptdirect"
super(CUOPTDirect, self).__init__(**kwds)
self._version = cuopt.__version__.split('.')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the problem line. If cuopt isn't actually available, then this panics. You should probably wrap this in a try block or move it into its own function, e.g.:

try:
    self._version = cuopt.__version__.split('.')
except DeferredImportError:
    self._version = None

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, upon chatting in the dev call more about this just now, this should probably just be moved into its own function, not at all in the init, because if it's in the init, it will import cuopt unconditionally no matter what.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support NVIDIA cuOpt solver

7 participants